Improved Algorithms For Keyword Extraction and Headline Generation From Unstructured Text

The problem of generating headlines for documents using purely statistical approach has been long standing. We describe here an improved extractive approach based on keywords. The insight here is that if one tries to summarize a document, one will invariably use keywords from the document itself. There are two aspects to the problem namely, finding the relevant set of keywords and finding the proper way to combine these words to reflect a coherent and grammatical headline. Keyword selection can be tackled using various statistics about keywords in the document, while the problem of generating the headline sentence from the selected keywords cannot be best solved without resorting to the knowledge about the semantics of the keywords. To enhance the accuracy of the headline in reflecting the content, the gist or topic is first identified. The results show that our extractive approach is feasible for generating informative headlines.